Unloading and loading a ClassiX® database

Unloading and loading a ClassiX® database

1. objectives and procedure

2. calling up the programmes

2.1 Call parameters of the programme CXUOSR.EXE (Unloading)

2.2 Call parameters for CXCOSR.EXE (checksum calculation)

2.3 Call parameters of the programme CXLOSR.EXE (load)

2.4 Example - a batch file

3. clustering

4. database analysis

5. cross-reference of the objects

1. objectives and procedure

The database is written to sequential files in ASCII format (DATABASE.DMP, FREEOBJ1.DMP, ...), which are in principle also readable for humans. They contain all information about the objects in the database, which is necessary to reconstruct these objects in a new database with the same data and to reconstruct all relations between the objects. Thereby the distribution of the objects to sub-databases, segments and clusters can be changed - this is usually the aim of such an operation.
Unload - Load is also an alternative to the schema evolution provided by ObjectStore (conversion of the database to a new schema, i.e. modified class definitions). During unloading the following information is lost:

• Objects that can no longer be reached (neither via a root entry point collection nor via a relation that starts from at least one other reachable object
• Indices via collections (including the index manager)

Logically deleted objects, however, remain as long as they are still referenced directly or indirectly by an active object.

The overall procedure - including checksum generation to ensure logical integrity - comprises four sub-steps:

• Unloading with programme CXUOSR.EXE
  In phase 1 all objects in the root entry point collections are unloaded. In phase 2 it is determined for each previously unloaded object whether relations originating from this object refer to an unloaded object, which is then unloaded. Since such objects also undergo the test just mentioned, the procedure is transitive and at the end of phase 2 all objects accessible via any access path are unloaded. For this purpose a hash table of the (database) addresses of all objects already unloaded is required (TMPFILE.DMP). The objects unloaded in phase 1 can be found in file DATABASE.dmp, the objects from phase 2 in files FREEOBJ1.dmp, FREEOBJ2.dmp, ... For the classes CX_BITMAP, CX_COM_OBJECT and CX_MULTIPLE_COM_OBJECT there is another file per object, which contains the data stream of the object.
  Phase 3 is optional: here the assignment of the objects in the root entry point collections is saved in the file REP_COLLS.dmp.
• Checksum the "old database" with CXCOSR.EXE (parameter -1)
  All objects in the database are read segment by segment and evaluated with a checksum. When reading the segments, even inaccessible objects are "seen". With the hashtable, which was built up during unloading, they can be distinguished from accessible objects.
• Loading the new database with CXLOSR.EXE
  Restoring the database also takes place in two phases. In phase 1 all objects are created and the data fields are assigned the corresponding values. Phase 2 is responsible for the reconstruction of the relationships between the objects. In order to make this possible, a hash table with the database addresses of all objects is created in phase 1 - similar to unloading - but here each entry also contains the database address of the new database: references in the "old database" can thus be translated very quickly into references in the "new database".
  The classification of the new objects in the Root-Entrypoint-Colelctions can be controlled by the description in the CLASSIX.INI-File - and thus can be changed compared to the unloaded original database - or can be restored identically by means of the information saved during the unloading in phase 3.

• Checksum of the "new database" with CXCOSR.EXE (parameter -2)
  A checksum is calculated for each object in the newly loaded database. Matching the checksum of the "old database" guarantees that the data fields of all objects have been restored correctly. In addition, the number of relations emanating from each object is evaluated, which also controls the recovery of the object relationship.
  In case of discrepancies, it is possible to view the objects concerned in both databases.

This is the procedure if phase 3 of the unload and load program is not used, i.e. the arrangement of the objects in the root entry point collections is determined by the specifications in CLASSIX.INI. The explicit insertion of the objects in phase 3 guarantees an absolutely identical assignment of the objects to the Root-Entrypoint-Collections (see procedure).

Note: Only if the total checksum in CHECKSUM.OLD and CHECKSUM.NEW are identical, the newly created database can be used instead of the unloaded "old" database.

2. calling up the programmes

All flags can start with / or -. Lower and upper case letters are equivalent. A parameter following a flag can be directly following or separated by blanks; for example:

cxuosr /iTS.INI /h5000000

or

cxuosr -I ts.ini /H 5000000

If successful, the programmes will provide the return code 0.

2.1 Call parameters of the programme CXUOSR.EXE (Unloading)

The two phases

• Unloading objects from the root entry point collections
• Unloading of all those objects that can only be reached via previously unloaded objects

run automatically one after the other if neither /1 nor /2 is specified.
When unloading, it is determined how the objects are grouped in clusters in the new database to be set up.
Phase 3 collects the information about the arrangement of the objects in the root entry point collections.

Phase 1 is restartable. Prerequisite: Hash table is stored in the memory (parameter /E or /F).
If a class has been completely unloaded after a specified time interval, the hash table is saved as file 'filename.nnn', with nnn = { 001, 002, ... }and a flag is written into the unload file (checkpoint).
At such a point you can put on again: Parameter /R. The resulting parts of the discharge data must be joined together at the end, eliminating superfluous information (data discharged between the last checkpoint and abort) - call with parameter /P.
The following overview shows all other flags relevant for restartability:

As 'restartInfo' the filename of the rescued hashfile (complete path) has to be entered (see example).

2.2 Call parameters for CXCOSR.EXE (checksum calculation)

The checksums for each individual object are written to the file OBJECTS.OLD at /1. If CXCOSR.EXE for the new database is run with /2, the file OBJECTS.NEW is still created, which contains all objects with their checksums. Checksum differences are marked in such a way that the objects concerned can be viewed in both the old and the new database with InstantView^® - e.g. with the object inspector - InstantView^®-Source: CHECKSUM.CXP. Further functions for analysing objects in the old and new database are provided by the CX_DB_UTILITY object.

The program CXCOSR.EXE provides a further function which is not directly related to the unloading of a database: the creation of an object cross-reference - i.e. an auxiliary database which can be used to determine for each object from which other objects it is referenced

2.3 Call parameters of the programme CXLOSR.EXE (load)

Here, too, both phases normally (neither /1 nor /2 specified) run automatically one after the other. CXLOSR.EXE generates a protocol in the file LOAD.LOG

The hash table (GATEWAY.DMP) forms the bridge between the old and new database, i.e. for each object location in the old database the location of the corresponding object in the newly loaded database is known. The table can be kept resident in memory during loading: locally (flag /F) or distributed to external servers (flag /E). A "distributed gateway file" can be combined to one file.

2.4 Example - a batch file

cxuosr -Its.ini
if ERRORLEVEL 1 goto EVENING
cxuosr -3 -Its.ini
if ERRORLEVEL 1 goto EVENING
cxcosr -1 -its.ini
if ERRORLEVEL 1 goto EVENING
cxlosr -its.ini -batch
if ERRORLEVEL 1 goto EVENING
cxlosr -its.ini -3
if ERRORLEVEL 1 goto EVENING
cxcosr -2 -its.ini
echo "Database sucessfully reloaded
break
exit
:EVENING
echo "Reload failed

3. clustering

With ObjectStore the data transfer between the database server and the clients is page-oriented. Therefore the arrangement of the objects in the database determines the performance of the applications to a high degree.

The clustering of the objects can be controlled for

• the object creation with InstantView® commands CreatePersObject, CopyPersObject
• the object creation when unloading/loading a database .

In both cases, clustering can be activated or deactivated in stages.

Under the aspect of clustering, an object is either a master or slave object:

A master object is stored in a segment of the database defined for its class; a cluster may be created for the master object. The information about the segment and any cluster to be created is described in the database layout (file CLASSIX.ODB).

If clustering is completely deactivated, there are only master objects - this is the standard assumption in the ClassiX® system.

The position of a slave object in the database determines another object already stored in the database to which it is connected. The creation of such an object with CreatePersObject, CopyPersObject must be delayed until a relation to an object already existing in the database is established. Therefore the mentioned InstantView® commands first create a (transient) object of the class CX_LAZY_CREATOR.
The procedure for unloading/loading a database is different and is described below.

Before doing so, however, the rules that determine when an object is master and when it is not will be discussed.

The clustering rules have been grouped into categories - according to pragmatic considerations - and, by assigning a bit to each category, a pattern is obtained with which the categories can be switched on and off independently of each other.

This pattern is set with the environment variable CX_CLUSTERING; the default setting is CX_CLUSTERING=0.

The following table shows the categories and the associated rules; in addition to the bit in column 1, a name was assigned in the second column.

Column 3 specifies which classes can appear as slave objects. While for elementary objects (e.g. the base classes CX_DATE, CX_VALUE ...) the class affiliation is sufficient as sole criterion, additional conditions are specified for more complex objects:

Dimension[0] = master

• A... means class A and all classes derived from it
• REP_pattern is the bit pattern that determines the classification of the object in the REP-Collection; either specified by the description of the class in CLASSIX.ODB or by explicit specification in CreatePersObject
• Conditions that refer to the master object master are only valid during unloading/loading, with CreatePersObject / CopyPersObject a CX_LAZY_CREATOR is always created
• If an object is created with CreatePersObject, the additional condition REP_pattern = 0 applies to all categories. A transient object CX_LAZY_CREATOR cannot be inserted into a REP collection.

The above rules should not only control the clustering produced by means of delayed object creation (CreateOersObject -> CX_LAZY_CREATOR) but should of course also be effective when unloading/loading a database.

The arrangement of the objects in the new database is already determined when unloading:

If a master object o is unloaded, all objects directly accessible by it via relations (pointers / collections) _or are tested according to the above rules, whereby objects o takes on the role of the master. _If there are slave objects among the objects, they are unloaded next and marked as belonging to o in the unload file.

When unloading the slave objects the same happens, the _or take over the role of the master and all objects accessible by them are tested to see if they can be "clustered" by _or. Also these indirectly reached objects get into the cluster belonging to object o. Object o then determines when loading the database how all objects subordinated to it are to be stored.

An object o "asks" all objects referenced by it whether they can be "clustered" by o.

Accordingly, it is important in which order the objects are unloaded.

During unloading, priority numbers (from 0 to 5) are assigned to the classes. In phase 1, all objects are unloaded from the in root entry point collections, with classes with low priority numbers coming first:

Within the same category the classes are unloaded according to the order of the class descriptions in CLASSIX.ODB, i.e.
also the order of the class statements in CLASSIX.ODB influences the clustering

4. database analysis

If the database contains even a single corrupt object, the unload process will generally terminate with a program error.

Program CXAOSR.EXE can detect such errors in an analysis run and write them to a report file:

/An%-m% bzw. /An%- oder /A-%m

/P iptFile1,iptFile2,...,iptFilen,outFile,xRefFile[,W]

Example: Distribute analysis runs for a large database on three PCs: cxaosr /A-33%, cxaosr /A34%-66%, cxaosr /A67%-

Note: In a batch file, remember the syntax of the parameters: double the percentage signs, e.g.: cxaosr /A34%%-66%%!

During the analysis (with flag /A...) the virtual function SanityCheck() is called for each object. A return code > 0 signals an error:

Slots

Objects

SanityCheck also calls itself up for members registered in the DDI, provided that the object in question is a property. In case of an error, the original error code + 10000 is returned.

Furthermore, relations to other objects are analysed (function CheckReferences):

Functions of the CX_DB_UTILITY object support an effective analysis of the error log.

5. cross-reference of the objects

Often it would be of interest to know for a certain object o all objects _or which reference the object o, i.e. all objects _or which have a pointer to o or a collection with o as element.

This information is kept in objects of the class CX_XREF. Building a cross-reference means that for each object exactly one CX_XREF object is created, which refers to all objects that reference the object represented by the CX_XREF object.

It makes sense to keep the CX_XREF objects in a separate physical database - after all, they are only a snapshot that will soon lose its relevance. CX_XREF objects refer to "normal" objects - but not vice versa. The physical sub-database for the cross-reference can therefore be deleted at any time without affecting the "actual" database.

The assignment of an object to "its" CX_XREF object is realised outside the database (in the gateway file).
The function GetObjectsReferringTo() of the object manager returns for an object o passed as parameter a collection of all objects _or that reference the object o.

For such an object o, to which no other object _or refers, no corresponding object of the class CX_XREF is created, unless you request with flag /O that a CX_XREF object is created for each object in order to find "lonely" objects in the database.

Description of an auxiliary database for the cross reference:

This description is added to the initialisation file (e.g. classix.ini) as long as you want to work with the cross-reference.

Note:
For performance reasons, the REP collection should be the CX_XREF object of type LIST.
For each object in the source database that is referenced by other objects (most of them) a corresponding object CX_XREF is created. Inserting them into a SET takes more and more time with a growing number of objects.

For large data volumes, a limit value for segment splitting should be specified for the object segment.

Objects of class CX_SLOT_XREF are created in the auxiliary database for the cross-reference of the dynamic data fields. Metainformation for object and slot cross-reference is shown in the following example.

The object-cross-reference is established by using

...

SET CX_XREFDB=myXReferenceDB.cxd

cxgosr -U -I modifiedIniFile.ini -batch

the partial database for auxiliary objects is created.

Afterwards, the cross-reference is built up with CXXOSR.EXE:

The function GetObjectsReferringTo() requires the Gateway file.
Gateway files (parameter /E) distributed to several external DCOM servers can therefore be combined with program CXXOSR.EXE:

The correct specification of the partial sizes (n1 + n2 + ... = ntotal) is compared with the specifications in the gatewayFilek files.

Sample batch files for building the complete cross-reference can be found here.

Partial cross-reference:
The construction of the object cross-reference is accelerated if only for a certain set of objects { o1, o2, ... } is determined, which objects _or one of the given objects { o1, o2, ... } are referenced. In the parameter /X a file is specified, which

• in the first line contains the estimated number n of objects and optionally the identifier EXCLUDE
• in the following lines the objects are specified by their location in the database

The prime number for the hash file is calculated from the number n - a larger n improves the performance.
EXCLUDE requires that the given objects are not tested as potential speakers. This is useful when repairing corrupt objects in the database: You want to know which objects refer to the "corrupt" objects, but exclude them from the test.

Example:

2000, EXCLUDE
<0|284|0|80f230|10000>, <0|284|0|80f350|10000>, <0|284|0|80f470|10000>, <0|284|0|80f590|10000>, <0|284|0|80f7e0|10000>, <0|284|0|80fa20|10000>, <0|284|0|80fb40|10000>, <0|284|0|80fc60|10000>, <0|284|0|80fd80|10000>, <0|284|0|810360|10000>, <0|284|0|8105bc|10000>, <0|284|0|8106ec|10000>, <0|284|0|810800|10000>, <0|284|0|810920|10000>, <0|284|0|810a40|10000>, <0|284|0|810b60|10000>, <0|284|0|810c80|10000>, <0|284|0|810da0|10000>, <0|284|0|810ec0|10000>, <0|284|0|81157c|10000>,
<0|284|0|811914|10000>, <0|284|0|811a84|10000>, <0|284|0|812040|10000>, <0|284|0|812268|10000>, <0|284|0|812320|10000>, <0|504|0|7d13cc|10000>

Commas as separators between the elements are optional.
The number of objects per line is unlimited as long as a line length of 512 characters is not exceeded.

Hide certain objects:
Individual objects can also be excluded from the complete cross-reference. This variant is mainly intended as a tool in case you have to deal with corrupt objects in the database. Structure of the file as shown above.

Database analysis and partial cross-reference together form a tool with which logically or physically corrupt or inconsistent objects in a database can be "repaired" (example procedure).

Note: By means of the cross-reference an object in the database can be replaced by another object (function Substitute of the DB-Utility object).